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PROPOSAL REVIEW AT NSF; 
PERCEPTIONS OF PRINCIPAL INVESTIGATORS 



I. Summary and Highlights 

This report presents the results of a survey conducted by NSF in 
late 1986 of more than 14,000 applicants whose proposals for 
research support had been awarded or declined during fiscal year 
1985. The principal purpose was to seek the views of individual 
investigators at academic institutions about NSF's competitive 
proposal review and decisionmaking process. Two- thirds (more 
than 9,500) responded. 

Nearly half the applicants were satisfied with the review 
process, but a substantial proportion (38%) were dissatisfied.^, 
Declinees (two--thirds of the applicants) were much more likely to 
be dissatisfied. This was particularly true of the 27% of repeat 
applicants whose proposals were consistently declined. 

/The reasons most often volunteered by dissatisfied applicants ^ 
were that the reviewers selected by NSF were not sufficiently 
expert in the subject matter of the proposal, or that the reviews 
were cursory, conflicting, or did not^seem to support the 
Foundation's decision./ Reasons such as cronyism, politics or 
biases of various types were cited less frequently. 

Whether awarded or declined, one-third of respondents felt that 
the reviewers' comments helped them to understand the 
Foundation's decision a great deal. Two-fifths of those who 
received awards said that the reviewers' comments influenced the 
course of their subsequent research. 

/Applicants' views varied according to their experience, i.e., 
depending roughly on the proportion of awards they received to 
proposals submitted./ Campus-based individual applicants fell 
into six "experience classes," ranging from those who submitted 
one proposal in five years and were declined, to those who 
applied many times and rece/ved awards every time. More than a 
quarter submitted proposals repeatedly but without success; they 
were the persons most likely to be dissatisfied and to believe 
that decisions to decline their proposals were made unfairly. 
The smaller group (13%) who consistently received funding for 
their proposals were much more satisfied with the review system. 

^Applicants' experiences, actions and views also differed 
according to which NSF research support division handled their 
proposal (sX/ in addition to considerable variation in award 
rates, the^iature and frequency of pre- and post-review contacts 
differed in important respects by division. / 
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Investigators who wrote, telephoned or visited a program officer 
before submitting a proposal were somewhat more likely to receive 
an award* (The survey did not address the reasons, but 
experience shows that in some cases investigators decide on the 
basis of such contact not to submit a proposal, and in other 
cases are able to sharpen a proposal before submitting it.) 

Over a quarter of those whose proposals were declined 
subsequently resubmitted them to NSF, in most cases with 
substantial revisions; a fifth submitted them to another funding 
source; and nearly half indicated that they had taken no furthei 
action by the time of the survey* 

Persons declined on their first proposal were less likely to 
contact a program officer for an explanation of the decision, 
less likely to resubmit revised proposals, and more likely to 
take no further action, than other types of applicants. Those 
who had previously received several awards were much more likely 
to make follow-up contact and to resubmit their proposals. 

Other results of the survey include: 

— 17% or more of the applicants to engineering and computer 
research programs, and 9% of all applicants, were of Asian 
background; 2% were members of other minority groups; 

— for 29% of applicants (51% of women), the proposal that was 
the object of the survey was the first they had submitted to NSF 
as an independent principal investigator; 

— 20% were competing for further funding of current NSF- 
supported work, while 80%, some of whom had current NSF funding, 
said their proposal was for new work; 

— 29% indicated that their proposal was for multidisciplinary 
research; 

— 75% of all applicants, and 97% of consistently successful 
ones, had served as a reviewer or panelist for NSF at least once 
in the five years before the survey; 

— the proportion of proposals being resubmitted following an 
earlier declination was 29%; 

— for many divisions, and for the Foundation as a whole, the 
award rate for resubmittals (27%) was less than that for first 
submittals (36%). 

The report also contains the first cross-tabulation of 
applicants' fields of research to the Foundation's research 
support divisions, and the Initial chart of award rates* by 
division for first proposals and resubmittals. 
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II. Introduction 

In November 1986^ the K tional Science Foundation mailed a 16- 
page^ 43--question survey to 14^282 scientists and engineers whose 
competitively-reviewed proposals for research grants had been 
awarded or declined by the Foundation during fiscal year 1985 
(October 1^ 1984 - September 30^ 1985). The purposes were to 
characterize the "traditional" applicant population, to explore 
their attitudes about the Foundation's proposal review system, 
and to seek their views about several matters relating to program 
operations . 

This report describes the applicant population in general terms 
and by six categories of proposal/award experience, characterizes 
their proposals and awards, and summarizes their views of the 
Foundation's proposal review system. 

The survey was targeted toward individual investigators at 
academic institutions. Several types of proposers were not 
surveyed: those who withdrew their proposals before a decision 
was made; those who applied through the small business program; 
those- requesting support for centers, facilities and similar 
organizations; and applicants to programs that employ non- 
standard review processes (e.g., the Presidential Young 
Investigators program). Investigators who received decisions on 
more than one proposal during FY 1985, about one-seventh of the 
targeted group, were asked to report on the proposal that was the 
subject of the latest decision. 

Two- thirds, or more than 9,500, survey recipients replied by 
January 9, 1987, when coding of responses was ended. About 88% 
of survey recipients who received grants in FY 1985 responded, 
and 52% of those whose proposals had been declined. All of the 
data on which this report is based has been adjusted to account 
for the disproportionate response rates by decision and NSF 
division so as to reflect the entire target population. 

The process for handling survey responses was designed to protect 
the anonymity of individual replies. Questionnaires were mailed 
by a survey research firm which also collected the responses, 
coded unstructured narrative comments, and created the data base. 
The resulting data was supplied to NSF's Program Evaluation Staff 
for analysis and reporting purposes. No attempt was made to link 
responses to particular persons or proposals. 
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The survey form is reproduced in Appendix G, with percentages of 
answers marked beside each question. Much of the analysis in 
this report was derived by combining various responses. More 
detailed information is available upon request (Note 1). 

A full understanding of the results of the survey requires a 
substantial degree of familiarity with the proposal review and 
decisionmaking systems employed by the various research 
divisions. Readers who are not entirely familiar with the review 
processes are referred to the description in Appendix A. The 
organizational structure of the Foundation's research support 
divisions included in this study are shown in Appendix B. 

III. The Applicant Population in General Terms 

At various points in the questionnaire, applicants were asked to 
check categories indicating their sex, race, and type of academic 
institution, and to state their field of science or engineering 
and year of highest degree. On the basis of self -reported data, 
the population of individual investigators whose proposals were 
awarded or declined during FY 1985 (Note 2) was as follows: 

(1) Sex: 87% were men, 13% women. 

Women were more numerous (14% or greater) among applicants 
to eight divisions, including all five divisions of the 
Biological, Behavioral and Social Sciences Directorate as 
well as the Information Science and Technology, Ocean 
Sciences and Research Initiation and Improvement divisions. 
They were least numerous (8% or fewer) in four of the five 
Engineering divisions, three of the four Geosciences 
divisions, and four of the five divisions of the 
Mathematical and Physical Sciences Directorate. 

(2) Race: 89% were white, 9% Asian, and 2% members of other 
racial/ethnic groups. 

Asian applicants were most numerous (17% or greater) in the 
five Engineering divisions and the Division of Computer 
Research. They were least numerous (6% or less) among three 
of the four Geosciences divisions and the Astronomical 
Sciences, Chemistry, Behavioral and Neural Sciences, Biotic 
Systems and Resources and Social and Economic Sciences 
divisions. 

Proposal submission by non-Asian minorities was small (less 
than 2%) across the Foundation, but relatively higher (3% or 
more) in three Engineering divisions, the Information 
Science and Technology Division and the Research Initiation 
and Improvement Division. 

(3) "Professional age": 21% of applicants had received their 
highest degree since 1980, 41% received it between 1970 and 1979, 
26% between 1960 and 1969, and 11% before 1960. 

4 



(4) Institution: 23% had submitted their proposal (s) through one 
of 21 academic institutions listed on the survey form that 
received 60 or more competitively reviewed individual - 
investigator research grants from NSF in FY 1985 (Note 3); 62% 
had submitted through another Ph.D. -granting institution; and 11% 
through a primarily undergraduate institution. The remaining 4% 
indicated "other" (note 4). 

(5) Education: 45% received their highest degree from one of the 
21 named institutions, 54% from another Ph.D. -granting 
institution, and 1% from a primarily undergraduate institution. 
22% had received their B.S. or B.A. degree from one of the 21 
institutions, 49% from another Ph.D. -granting institution, and 
29% from a primarily undergraduate college or university. 

(6) Reviewer/Panelist: 75% of the surveyed group had served as a 
mail reviewer or panelist for NSF at least once in the five years 
before the survey. The lowest percentages (<66%) were in four 
divisions of the Engineering Directorate, and the highest (>90%) 
in three divisions of the Geosciences Directorate. 90% of 
awardees, and 67% of declinees, had some experience as a reviewer 
or panelist. Younger investigators, and those from primarily 
undergraduate institutions, were much less likely to have had 
such experience. 

(Note: Although comparable data from previous yeai - is not 
available, the FY 1985 applicant population undoubtedly included 
more .first-time women applicants; the Research Opportunities for 
Women program, initiated in FY 1985, received 404 proposals. 
Moreover, special efforts to attract applicants froui primarily 
undergraduate institutions began the previous year and were 
expanded in FY 1985.) 

IV. Proposal Characteristics and Award Rates 
A. Proposal Characteristics 

Proposals from the FY 1985 applicant population can be 
characterized as follows: 

(1) Field of research and NSF program structure: Respondents were 
asked to name the program or division to which they applied (Q.6 
of the survey) and to indicate their principal field of research 
(Q. 30). We categorized the program information using the set of 
research support divisions that existed in FY 1985 (note 5), and 
categorized fields using a modified version of a standard 
classification (note 6). The resulting table (Appendix C) is the 
first systematic look by program division at the research fields 
of campus-based individual investigators. 

For some fields, such as astronomy and mathematics, there is 
a very close correspondence with divisional structure; but 
significant numbers of investigators in chemistry and 
physics applied across the Foundation. 
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The last line of the table shows the relative volumes of 
proposals submitted by the surveyed population, in general, 
divisions in the Biological, Behavioral and Social Sciences 
Directorate and the Mathematics Division received two to three 
times the number of individual-investigator proposals as 
divisions in the physical and geophysical sciences, engineering 
and computer and information research. (As noted earlier, the 
survey excluded applicants representing centers, facilities and 
large groups — who are more likely to apply to the latter 
divisions ) . 

(2) New research vs. ongoing work (Q. 1 of the survey): 80% of 
respondents indicated that their proposal was a for research 
project; not being funded by NSF at the time of submittal; 20% 
were competing for further funding of NSF-supported research. 

Proposals for support of not-current ly- funded work were more 
likely to be from younger applicants, women, persons 
resubmitting previously declined work, and those from 
primarily undergraduate institutions. 

(3) New proposal vs. resubmittal (Q. 2): 29% of proposals decided 
upon during FY 1985 were resubmittals of previously declined 
proposals. 

The highest resubmittal rates (greater than 40%) were to the 
Biotic Systems and Resources, Earth Sciences and Ocean 
Sciences Divisions. Resubmittal rates of 25% or lower were 
found in ten cd visions, with the lowest rates in the Design, 
Manufacturing and Computer Engineering, Information Science 
and Technology, Atmospheric Sciences, Mathematics and 
Physics Divisions. 

25% of proposals from women, and 29% of proposals from men, 
were resubmittals. There was very little variation by race 
or by year of highest degree. Applicants at primarily 
undergraduate institutions were slightly more likely (32%) 
to be resubmitting than those at doctoral institutions. 

(4) First-time principal investigator (Q. 5): Overall, 29% of 
applicants said they were submitting their first proposal to NSF 
as independent principal investigators. 

26% of the men and 51% of the women applicants were doing so 
(FY 1985 being the first year of a program for women 
investigators without prior support.) As expected, 
investigators who received their degrees mostreceritly were 
more likely to be first-time Pi's: 59% of those whose 
highest degrees were awarded since 1980 ( "young 
investigators"); 29% of those in the 1970-79 cohort; 14% in 
the 1960-69 cohort; and 10% of those awarded their highest 
degree in 1959 or before were submitting for the first time 
as principal investigator. 
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(5) Multidisciplinary research (Q. 4): 29% indicated that cheir 
proposal was for support of multidisciplinary resisarch. 

Half of the proposals to the Division of Polar Programs were 
multidisciplinary; other divisions handling more than 37% of 
such proposals were Ocean Sciences, Information Science and 
Technology, and Social and Economic Sciences. 

Divisions with 21% or fewer proposals described as 
multidisciplinary were Cellular Biology, Computer Research, 
Astronomy, Chemistry, Materials Research, Mathematics and 
Physics. 

Very little difference was reported by first submittal vs. 
resubmittal, and no difference by sex. Applicants from 
undergraduate institutions and young investigators were 
slightly less likely to report their research proposal as 
multidisciplinary. 

(6) Character of Research (Q.20): 53% of awardees agreed in the 
statement that "all or much" of the research being carried out 
under their award has "applied, practical or policy implications 
beyond the advancement of knowledge." 

More than 75% of applicants to the five engineering 
divisions and two computer-related divisions characterized 
their research in this mam, or, compare'' with 23% of awardees 
in Astronomy and Physics. 

Such implicacions were reported more frequently by young 
investigators, one-time awardees, persons with no reviewer 
experience, and Asian and other minority investigators. 

Of the half of respondents to the above questions who 
answered a subsidiary question (Q.20A), about a quarter 
indicated that such implications were clear and immediate, 
half indivHted they were fairly clear but long range, and a 
quarter feit that potential implications of this sort 
existed but were not clear. 

(7) Amounts requested (Q. 8): Respondents reported the amount 
requested in their proposal on a yearly basis (to the nearest 
thousand dollars including indirect costs) in six categories: 

(Dollars in thousands) <30 30-49 50-69 70-99 100-149 >150 

% or proposals 13% 21% 22% 23% 14% 7% 



% of awards 



16% 20% 20% 23% 15% 7% 



Amounts requested differed considerably; for example 61% of 
Mathematics proposals, and 24% of those to the Social and 
Economic Sciences Division, requested less than $30,000; at the 
other end of the scale, 24% of proposals to the Physics Division 
and more than 11% of proposals to four other divisions requested 
more than $150,000. Younger investigators and those from 
primarily undergraduate institutions tended to request smaller 
amounts of money. The Foundation-wide distribution of awards by 
amount requested was almost the same. (Note: Amounts requested 
are generally higher than actual award amounts). 

B. Award Rates by Division 

Award rates by division are displayed in Appendix D. Chart (a) 
of the Appendix portrays "raw" award rates, i.e., FY 1985 awards 
divided by the sum of all award and decline decisions; these were 
calculated for the target population from NSF records (and used 
to adjust for the disproportionate number of survey responses 
from awardees). 

For the Foundation as a whole, 34% of proposals from the surveyed 
population were awarded. Rates varied considerably by division: 
generally lower for the life sciences, social sciences and 
engineering divisions; and higher for the geosciences, computer 
sciences and physical sciences divisions. 

By asking whether the proposal that respondents were reporting on 
was awarded or declined (Q.15) and whether it was a resubmittal 
(Q.2), we were able to construct the following profile: 

(1) on first submittal (chart b): overall, 36% were funded. 
Again, rates varied considerably by division. 

(2) on resubmittal (chart c): overall, 27% were funded. Rates 
varied by division, but not as much as with first submittals. 

(3) eventually funded (chart d): the NSF-wide percentage of 
"traditional" research proposals eventually funded could be as 
high as 48%. The chart reflects this estimate, which was 
calculated by assuming that resubmittals are as common as the 
responses to Question 2 indicate. Question 15A suggests they mav 
not be - (See VI. C below). 

C. Award Rates by Other Proposal Characteristics 

(1) Multidisciplinary proposals were awarded across the divisions 
at a rate of 30%. One division (Design, Manufacturing and 
Computer Engineering) awarded them at a rate 6% higher than is 
average rate, while a difference of 10% or more in favor of 
single-disciplinary proposals existed in the Computer Research, 
Information Science and Technology, Atmospheric Sciences, Polar 
Programs and Physics Divisions. 
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(2) First-time principal investigator: The Foundation -wide award 
rate was 29%, as compared with 36% for investigators applying for 
the second or later time. The rate for most divisions did not 
vary much from their division average. Two engineering divisions 
made awards to first-time investigators at higher rates than 
their division averlige and four (Information Science and 
Technology, Social and Economic Sciences, Earth Sciences and 
Physics) at somewhat lower rates. 

V. Six Categories (Experience Classes) of Applicants 

Recipients were asked to state (Q. 38) how many proposals they 
had submitted to NSF in the five years prior to completing the 
questionnaire (i.e., December 1982-1986), and how many were 
funded by the Foundation (Q. 39). 

Based on their proposal/award histories, we have defined and 
named six categories of applicants: 

(1) One-time awardee: 5% of applicants submitted one proposal 
during the five years preceding the survey, and it was awarded. 

(2) Frequent awardee: 26% of applicants submitted two proposals, 
one of which was awarded; or three or more proposals, two or more 
(but not all) of which were awarded. 

(3) Consistent awardee: 13% of applicants either received as many 
awards as proposals submitted (e.g., ranging from two proposals, 
two awards to five proposals, five awards) or received five or 
more awards over the five years, no matter how many proposals 
were submitted. 

(4) One-time declinee: 13% of applicants submitted one proposal 
during the five years, which was declined. 

(5) Frequent declinee: 14% of applicants submitted three or more 
proposals during the five years, only one of which was awarded. 

(6) Consistent declinee: 28% of applicants submitted two or more 
proposals during the five years, and all were declined. 

Characteristics of applicants by experience class are presented 
in Appendix E. Principal differences from the general population 
of surveyed applicants are as follows: 

(1) Of one-time awardees, 21% were women compared with 13% of 
applicants generally. More were from the "top 21" institutions 
and fewer from the "other Ph.D. -granting" institutions. 12% of 
applicants with one proposal/one award were from the oldest group 
(highest degree in 1959 or before) and 29% were young 
investigators (highest degree since 1980). 

7% or more of applicants to the following five divisions 
were one-time awardees: Social and Economic Sciences, 
Behavioral and Neural Sciences, Molecular Biology, Computer 
Research, and Information Science and Technology; one-time 



awardees were least numerous (less than 2%) among applicants 
to three divisions: Ocean Sciences, Polar Programs, and 
Emerging and Critical Engineering Systems. 

(2) Frequent awardees were slightly more likely than the average 
NSF applicant to be from one of the "top" institutions and 
slightly less likely to be "young investigators". 91% had served 
as a reviewer or panelist for NSF at least once in the five years 
before the survey. 

They were more numerous among applicants to the geosciences 
divisions and to the Biological Systems and Resources, 
Astronomical Sciences, Information Science and Technology, 
Electrical, Communications and Systems Engineering and 
Chemical, Biochemical and Thermal Engineering Divisions. 
The smallest proportions were among applicants to four 
divisions of the Biological, Behavioral and Social Sciences 
Directorate . 

(3) Consistent awardees were more likely than the average 
applicant to be male, older (19% with highest degrees prior to 
1960 and 35% with degrees received in the 1960s), and much more 
likely to be associated with a "top 21" institution. 97% had 
served as a reviewer or panelist. 

24% or more of applicants to six divisions were consistent 
awardees: Atmospheric Sciences, Polar Programs, Ocean 
Sciences, Astronomy, Mathematics and Physics; and 12% or 
fewer among applicants to all five divisions of the 
Biological, Behavioral and Social Sciences Directorate and 
four of the five Engineering divisions. 

(4) 25% of one-time declinees were women; 18% were from primarily 
undergraduate institutions and 17% from "top 21" institutions; 
29% were young investigators; and most (56%) had not served as a 
reviewer or panelist. 

One-time declinees comprised 15% or more of the applicants 
to six divisions, four of them in the Biological, Behavioral 
and Social Sciences Directorate, one in Engineeiing 
(Mechanics, Structures and Materials) and the Research 
Initiation and Improvement Division. This group comprised 
between 5% and 7% of applicants to the five divisions of the 
Mathematical and Physical Sciences Directorate and between 
2% and 6% of applicants to the four Geosciences divisions. 

(5) 45% of frequent declinees were in one age cohort (highest 
degree in the 1970s). 

The five Engineering divisions and the Divisions of 
Materials Research and Astronomical Sciences had the highest 
proportions (18% or more) of frequent declinees; and the 
lowest proportions (8% or less) were found among the 
Atmospheric Sciences, Social and Economic Sciences and 
Research Initiation and Improvement Divisions. 
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(6) Consistent declinees (the largest of the six classes of 
applicants) were more likely to be from "other Ph.D. -granting" 
institutions and less likely than the average applicant to have 
served as a reviewer or panelist. 43% received their highest 
degrees in the 1970s* 

More than 40% of the applicants to the Division of Cellular 
Biology were in this group ^ and more than 29% of applicants 
to seven other divisions: Behavioral and Neural Sciences^ 
Biotic Systems and Resources^ Molecular Biology^ Electrical^ 
Computer and Systems Engineering^ Mechanics^ Structures and 
Materials Engineering^ Chemistry^ and Mathematical Sciences* 
Consistent declinees comprised 20% or less of the applicants 
to the Information Science and Technology^ Atmospheric 
Sciences, Polar Programs, Ocean Sciences and Astronomical 
Sciences divisions. 

VI* Actions Taken by Applicants 

A. Before Submitting Proposals (Q. 7) 

As Appendix F shows, 52% of applicants telephoned someone at NSF 
to discuss a proposal before submitting it; 21% wrote, 13% 
visited the Foundation, and 4% submitted some form of preliminary 
proposal* 26% indicated that they knew the program officer prior 
to submittal* One-- third had received a mailing from the 
Foundation, and 5% heard or spoke with someone from NSF who had 
visited their institution. In one-quarter of the cases, the 
applicant's institution handled all the contacts* 

(1) a-l^e nature of the pre-proposal contact varied widely by 
program: applicants with more complex and costly proposals (e.g. 
in astronomy, atmospheric sciences, physics, oceanography, polar 
programs) were much more likely to have written a pre-proposal 
inquiry. More than a fifth of the applicants to engineering 
divisions visited NSF, but very few biologists wrote or visited. 
Half of the applicants to the Mathematical Sciences division had 
no personal contact with NSF (their institution handled all 
contacts ) . 

(2) The amount and type of pre-proposal contact also differed 
greatly by the six experience categories: Frequent and consistent 
awardees were much more likely to have written, called or 
visited, and much less likely to have permitted their institution 
to handle all the contacts, than members of the other groups. 

(3) Investigators who contacted NSF staff in some way prior co 
submitting a proposal were somewhat more likely to be funded than 
those who did not* (Note: The survey did not address the reasons 
for this outcome, but experience shows that in many cases 
investigators decide not to submit a proposal, and in other cases 
they are able to sharpen a proposal before submitting it). 

(4) The likelihood of an award was much higher for applicants who 
said they knew the relevant program officer prior to submitting 
(many of whom had been frequent or consistent grantees). 
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Applicants to five divisions were most likely (>40%) to have 
known the program officer before applying: Atmospheric^ 
Earth and Ocean sciences^ Astronomy and Physics; and least 
likely (<14%) were applicants to the Cellular Biology^ 
Molecular Biology, and Research Initiation and Improvement 
Divisions. 

33% of those who had served as a reviewer or panelist 
indicated that they knew the program officer, compared with 
7% of those who had not. Persons with reviewer /panelist 
experience were twice as likely to have visited, and 
slightly more likely to have called NSF before submitting. 

(5) Young investigators were about as likely as other age groups 
to have telephoned, written, or read an NSF mailing. They were 
much less likely to have known the program officer or visited the 
Foundation, and more likely to allow their institution to make 
all the pre-proposal contacts. 

(6) Applicants from undergraduate institutions were more likely 
to have written or telephoned, and much less likely to have known 
the program officer or to have visited NSF. They were also much 
less likely to have worked entirely through their institution's 
research support office. 

B. Awareness of Alternative Programs 

Three sets of questions (Q. 23, 24, 25) explored the degree of 
awareness about, and readiness to apply for support through, 
alternative channels for less-experienced women or minority 
investigators and for applicants of all sorts from undergraduate 
institutions (the programs of Research Opportunities for Women, 
Minority Research Initiation, and Primarily Undergraduate 
Institutions, respectively). 

The majority of eligible respondents for all three programs did 
not submit their proposals through these channels, principally 
because they did hot know about them at the time they applied • 

In all three cases, eligible applicants who had served as 
reviewers once or more for NSF were somewhat less likely than 
those with no reviewer experience to have applied through an 
alternative program. 

Applicants who were aware of, and eligible for, an alternative 
program but did not apply through it were asked to state their 
reasons. They most often volunteered that the specific program 
did not meet their needs, that their proposal fit better in 
another program, or that they expected the outcome to be more 
successful if their proposal were handled through regular 
channels. 

(Note: The PUI program and the ROW program were relatively new 
at the time of the survey. Some unknown proportion of women 
applicants whose proposals were decided upon in FY 1985 had 
submitted them before the ROW program began in that year. 
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More detailed analysis of this set of questions was not 
undertaken because the numbers of persons responding were small.) 

C. After Being Declined 

Two-thirds of the proposals from the surveyed population were 
declined. When declinees were asked what actions they 
subsequently took with their proposal (Q. 15A): 

— 48% indicated they took no further action (presumably from the 
time of the declination to the time when they responded to the 
survey, anywhere from 14 to 26 months); 

— 25% resubmitted it to NSF, most with substantial revision; 

— 11% submitted it to another funding source with little or no 
revision; 

— 7% were in the process of developing a revised proposal; and, 

— less than 2% each were conducting the proposed research under 
another grant, conducting it without funding, deciding what step 
to take with their proposal, or seeking further information about 
the decision ( answers could be multiple ) . 

(1) Again, there were large differences among applicants to 
various divis. ons . For example : 

10% of applicants to the Earth Sciences division, and 9% of 
those to Astronomical Sciences, said they resubmitted the 
declined proposal with little or no change, while in most 
divisions, less than 3% of applicants did so. Applicants to 
three biosciences divisions, two engineering divisions. 
Chemistry, and Materials Research were much more likely to 
have submitted the declined proposal to another agency, with 
or without revision. 

(2) First-time declinees were much less likely to resubmit to 
NSF, and much more likely to take no further action, than other 
experience classes. Frequent awardees (and frequent declinees) 
were much more likely to resubmit, generally (but not always) 
with revisions. 

(3) Applicants from various types of institutions and from 
various age groups did not show much difference in this regard. 

(4) Women applicants were a little more likely than men to revise 
and submit to another agency (Note that women are relatively more 
numerous among bioscientists, who are more likely to do so). 

(5) Minority applicants were less likely to have taken further 
action than Asian or white applicants. 
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Almost half the applicants indicated that they contacted the 
program officer to discuss the reasons for the declination 
(Q. 17). 

(1) Post-declination contact varied considerably by experience 
category: 72% of the one-time declinees did not ask the program 
officer for an explanation (i.e., beyond receiving the verbatim 
reviews and, where applicable, panel summaries); frequent and 
consistent awardees were much more likely to do so, 

(2) Those applying to the geosciences divisions were more likely, 
and those applying to the Mathematical Sciences, Social and 
Economic Sciences and Computer Research Divisions much less 
likely to have contacted the program officer. 

(3) More than half of the applicants who had served as a reviewer 
or panelist contacted the program officer compared with about a 
third of those who had not. 

(4) Minority applicants, women and applicants from undergraduate 
institutions were less likely to contact the program officer. 

(5) Persons resubmitting a previously declined proposal were more 
likely to contact the program officer after it was for the second 
time declined. 

82% of declinees said in response to Q. 18 that they were unaware 
of the Foundation's formal reconsideration procedure (explained 
in Appendix A). Percentages did not vary greatly among 
divisions, but they did range from higher to lower across the six 
experience categories, and with advancing "professional age"; 
i.e., the less experienced the applicant, the less he or she was 
likely to be aware of the possibility. 

VII. Opinions About the Proposal Review System: 

A. Overall Satisfaction (Q. 22) 

Applicants were aLked to indicate on a five-point scale how 
satisfied or dissatisfied they were with NSF's review process. 
Overall, 49% reported being satisfied or moderately satisfied, 
14% were neutral, and 38% were dissatisfied or moderately 
dissatisfied. 

(1) Four-fifths of awardees and one-third of declinees were 
satisfied; half of the declinees were dissatisfied V7ith the 
review process. 
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(2) By experience: As the following table Indicates, applicants* 
levels of satisfaction ranged In order of experience class from a 
low of 27% among consistent decllnees to a high of 87% among one- 
time awarders: 



Satisfied Neutral Dissatisfied 



Consistent Decllnee 27% 16% 57% 

One-time Decllnee 35% 21% 44% 

Frequent Decllnee 40% 16% 44% 

Frequent Awardee 61% 11% 28% 

Consistent Awardee 83% 5% 12% 

One-time Awardee 87% 7% 6% 



(3) By division: The highest levels of satisfaction were among 
applicants to the Atmospheric Sciences (61%), Physics (58%) and 
Social and Economic Sciences (56%) Divisions; the lowest levels 
(from 34% to 43%) were among applicants to the five Engineering 
divisions • 

(4) By review method: Levels of satisfaction did not differ 
significantly with respect to applicants to divisions that used 
panel review compared to divisions that employed mall review 
only. 

(5) Resubmlttal: 48% of persons resubmitting declined proposals 
were dissatisfied compared with 33% of applicants not doing so. 

(6) By Institution: Applicants from "top" Institutions were 
slightly more likely to be satisfied, and those from primarily 
undergraduate institutions more likely to be dissatisfied. 

(7) By "professional age": Not much difference, except that 
young investigators were a little more likely to be neutral. 

(8) By sex: Almost no difference. 
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When respondents indicating dissatisfaction wore asked to write 
in reasons, 92% wrote one or more. The reasons they volunteered, 
in order of frequency, were: 



reviewers or panelists not expert in the field, 

poorly chosen, poorly qualified 18% 

— reviews were perfunctory, cursory, non-substantive 17% 
reviews were conflicting 12% 

— cronyism, politics; "old boyVs network" 12% 

— decision was unclear or inconsistent with the reviews 10% 

— reviews contained remarks considered biased, 7% 
personal or intemperate 

— process is "unfair" (not further defined) 7% 

— process is too slow 5% 

— system biased against "innovative" proposals 5% 

— system biased against particular research topics 5% 

— program officer unqualified or inexperienced 5% 

— system biased toward "big schools" 4% 

— system biased against interdisciplinary proposals 3% 

— proposal handled by wrong program 2% 



(1) Awardees and declinees gave similar reasons in about the same 
order, except that 13% of dissatisfied awardees said that the 
review process was too slow, and dissatisfied declinees were 
slightly more likely to cite politics or cronyism. 

(2) Nearly three times as many consistent declinees replied to 
this question as other types of applicants, and they were 
slightly more inclined to cite politics or cronyism. 

(3) 10% of dissatisfied applicants from Primarily Undergraduate 
Institutions cited bias toward "big schools" as a source of their 
dissatisfaction, compared to 3% from doctoral institutions that 
were not in the "top 21". Dissatisfied PUI applicants were also 
a little more likely to cite politics or cronyism: 14% listed 
this as their first or second reason compared to 12% overall. 

(4) Younger investigators were more likely to cr.te cursory 
reviews, and older investigators more likely to say that some 
reviewers had the wrong scientific background or were poorly 
chosen by the program officer. 

(5) Applicants of Asian background were more likely to cite 
cursory reviews or poorly chosen/poorly qualified reviewers, 
while minority applicants were more likely to cite conflicting 
reviews, politics/cronyism or bias/personal remarks. 

(6) Again, responses differed by division: One-fifth of 
applicants to the Division of Mathematical Sciences cited 
politics/cronyism compared with 4% of those to the Division of 
Polar Programs. But, as noted earlier. Mathematics applicants had 
the least pre-proposal contact of all NSF applicants and were 
twice as likely as the average applicant to allow their 
institutional research office to handle all interactions with the 
Foundation. The reverse was true of Polar Programs applicants. 
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In addition; 5% of dissatisfied respondents said NSF should offer 
a formal opportunity to rebut reviewers* remarks before the 
award/decline decision is made. This was pexticularly true among 
applicants to the Divisions of Mechanics, Structures and 
Materials Engineering, Polar Programs, Astronomy, Chemistry, 
Physics and Materials Research. 

B. "Fairness" of Decisions to Decline 

Question 16 asked declinees zo indicate whether they thought the 
decision to decline their proposal was made fairly or unfairly; 
60% thought the decision was "unfair". 

(1) Among experience groups, 64% of consistent declinees, 63% of 
frequent declinees and 54% of one-*time declinees held this view. 

(2) The highest levels of perceived unfairness (65% or higher) 
were found among applicants to tv/o of the four engineering 
divisions and to t le Materials Research, Physics, Computer 
Research and Information Science and Technology divisions. The 
lowest levels (56% or lower) were among applicants to the Social 
and Economic Sciences, Behavioral and Neural Sciences, 
Oceanography and Chemistry Divisions. 

(2) 68% of resubmitters thought the decision to decline their 
resubmitted proposal was unfair. 

(3) The tendency to report unfairness increased with professional 
age; for example, 69% of those who received their highest degree 
in 1959 or before, compared with 56% of young investigators. 

(4) Slightly more applicants from "top" institutions were likely 
to report the declination as unfair, and men were slightly more 
likely to so report. 

(5) There was little difference among white and minority 
applicants, but applicants with Asian backgrounds were far more 
likely to indicate an unfair decision. 

When those who indicated the declination wcis made unfairly were 
asked to state reasons, 58% of them did so. Their reasons were 
given about as often, and in about the same order, as the answers 
to the more general question about overall satisfaction. In 
addition, 4% of respondents to this question said that less 
important research than their proposal was funded, and 3% 
attributed the unfairness to "insufficient program funds". 
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C* Usefulness of Reviewers' Conunents 

Respondents were asked (Q. 21) how much the reviewers' comments 
helped them to understand the Foundation's decision. One-third 
said they helped a great deal, one-third found them somewhat 
helpful, and one-third said they were slightly or not helpful. 
Two-fifths of the declinees reported (Q. 15B) that the reviewers' 
comments substantially influenced their decision regarding the 
declined proposal, and another third said that the comments 
slightly influenced their decision. 

More than two-fifths of the awardees reported (Q. 19) that the 
comments had influenced the research they subsequently conducted. 



D. Views on Five Specific Statements (Q. 27) 

Respondents were asked to what extent they agreed or disagreed 
(on a five-point scale) with five statements frequently made 
about the NSF proposal review system. Column one below indicates 
the percentage that agreed and column two the percentage that 
disagreed with each statement: 



(A) (D) 

(a) (NSF's) review process makes it difficult for 
researchers to submit multidisciplinary proposals 39% 20% 

(b) NSF should make greater use of grants that 
permit groups of researchers to band together in 

pursuing common research problems 39% 29% 

(c) NSF is not likely to fund high-risk exploratory 
research because the likelihood of obtaining 

favorable reviews is slim 67% 13% 

(d) In making awards NSF should place substantially 
more emphasis on the research history of the 

investigator, except for young researchers 49% 26% 

(e) Proven researchers experience difficulty obtaining 
new awards when they apply for grants outside their 
disciplines 52% 8% 
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Awardees and decllnees tended to have different levels of 
agreement with the various statements. The highest level of 
agreement overall was with statement (c), but this was also the 
one about which awardees and declinees were most split, with 55% 
of awardees agreeing, compared with 74% of declinees. 

Applicants who had some experience as reviewers for NSF were more 
likely to disagree with statements (a), (b) and (c) while non- 
reviewers were more likely to disagree with statement (d). 

Applicants from undergraduate institutions were more likely to 
disagree with statements (d) and (e), while applicants from both 
groups of doctoral institutions were more likely to disagree with 
statements ( a ) and ( b ) . 

Older applicants were much more likely to agree, and younger 
applicants much more likely to be neutral about, statement (e). 
Older applicants were also much more likely to agree, and younger 
applicants more likely to disagree, with statement (d). 

E. Preference for NSF vs. Other Funding Source 

Applicants were asked (Q. 42) which Federal agency or other 
funding source is their first choice for support of their 
research. As might be expected, 66% cited NSF. Other agencies 
most often cited were The National Institutes of Health (15%), 
the Department of Defense and its various components (6%), and 
the National Aeronautics and Space Administration (2%). 

(1) 79% of awardees and 59% of declinees indicated that NSF was 
their first choice for support. 10% of awardees and 18% of 
declinees cited NIH. 

(2) The highest preference for NSF (92%) was among applicants to 
the Mathematical Sciences Division. Twelve other divisions had 
preference indicators of 75% or higher. For two divisions. 
Cellular Biology and Molecular Biology, more than half cited NIH 
as their first choice, as did 35% of the applicants/to the 
Behavioral and Neural Sciences Division and 22% of^^those who 
applied to the Chemistry Division. 

(3) DOD support was preferred by 10% to 21% of applicants to 
seven divisions, principally in areas of engineering, computer 
science and materials research. 21% of applicants to Astronomical 
Sciences and 12% of those to Atmospheric Sciences preferred NASA. 

(4) 62% of the one-tlir»e declinees and 44% of the consistent 
declinees preferred othei* funding sources (principally NIH); 69% 
or more of the applicants in the other four experience categories 
preferred NSF. 
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When respondents were asked why the funding source they named was 
their first choice, 84% did so. The reasons they volunteered were 
as follows (in order by comments about NSF): 

Percent Preferring; 



Agency supports basic research 
Supports research "in my area" 
Only funding source for my work 
Flexibility in research goals 
Fairer review process 
More prestigious source 
Fewer "strings" on the work 
Prior success w/ funding source 



XTC T? 

NSF 


DOD 


NIH 


NASA 


Other 


31% 


4% 


3% 


7% 


4% 


26% 


27% 


28% 


45% 


43% 


13% 


2% 


1% 


2% 


2% 


10% 


3% 


2% 


3% 


3% 


7% 


13% 


14% 


6% 


7% 


6% 


(others less than 


I 1%) 


5% 


3% 


1% 


2% 


1% 


5% 


10% 


10% 


10% 


12% 



Eleven other reasons were cited more than 1% of the time. 

VIII* Excerpts from Selected Narrative Statements 

The last question (Q. 43) on the survey form encouraged 
respondents to provide "any comments that you might have 
regarding the selection process of proposals as it operates at 
NSF." Half replied to the open-en^ed invitation, with comments 
ranging from one sentence ( "Thanks'" for the money!") to several 
typewritten pages. Coding of the narratives was attempted but 
resulted in too many categories for analytic purposes; many 
respondents addressed several topics, while somo placed them in 
the context of a narrative about their research career and their 
personal history of interaction with the Foundation. 

The following 34 excerpts have been chosen to represent the 
diversity of issues raised in the narrative statements and to 
illustrate some of the points that resulted from analysis of the 
survey data. 

(1) "NSF works about as well as can be expected. . .With a few 
exceptions. . .good science gets funded and mediocre science 
doesn't, which is all I ask." 

— Frequent awardee with extensive reviewer experience. 



(2) "Totally unfair as practiced by some program managers. They 
like to turn down proposals with high ratings as a justification 
for requests of increased budgets." 

— Consistent declinee, little reviewer experience. 
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(3) "The peer review system, though flawed, is still the best, 
provided program directors make the effort to select true 'peers' 
(not easy, I*m sure). NSF is commonly perceived to be subject to 
'good-old-boy* reviewing and as such, t is hard to cor.vince 
unsuccessful applicants, especially younger ones, that their 
proposals were not funded simply because they didn'^t belong to 
the 'club.'" 

— Frequent awardee, experienced reviewer. 

(4) "The peer review system of NSF is the most rigorous and 
fairest system operative among government agencies. The worst 
systems are the 'old boy' networks of DOD, the Bureau of Mines 
and other agencies whose review system is largely internal. The 
weakest part of the N3F system is the panels, and I think that 
the system of panels should be abolished wherever possible..." 

— Frequent declinee with some reviewer experience. 

(5) "No system of reviewing is without faults. However, a 
system where all proposals in a given area are reviewed in common 
by the same panel, like that used at NIH or DOE, would at least 
lead to more consistent results.'' 

— Frequent awardee, little reviewer experience. 

(6) "Like many colleagues I have given up for now hope to get 
support from NSF. i. Compared with NIH reviews, with which I am 
familiar, the NSF reviews are superficial, sometimes incompetent, 
and the necessary balance between confidentiality of the review 
process and accountability of the reviewers is skewed too much 
toward the former; in other words, it is too easy for the 
reviewer to sink a proposal. The reviewers take a scientific 
disagreement with the ideas in the proposal as a reason for poor 
rating . " 

— Frequent declinee, no reviewer experience. 

(7) "Process works better than that used by NIH. Use of a 
larger pool of ad hoc reviewers makes for a less parochial view 
of proposals." 

— Frequent awardee, experienced reviewer. 

(8) "The only problem I have had is the feeling that the 
reviewers were inappropriate for the proposals... This happens 
to have occurred in cases where, in my opinion, the proposed 
research was somewhat novel and probably did not fit into the 
particular program to which it was assigned. The reverse 
situation has been about as frequent — the reviewers knew more 
about the proposed work than I did... There have also been cases 
where I disagreed with the conclusions of a review panel but two 
independent panels (at NIH and NSF) had the same objections." 

— Frequent awardee, some reviewer experience. 
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(9) I revised my proposal based largely on earlier reviewers' 
comments, and re-submitted a better proposal. That proposal was 
also turnea down, but the reviewers' comments contained 
criticisms that weren't even mentioned by the first set of 
reviewers. Had they been mentioned the first time, I would have 
worked on revising those points also." 

—Consistent declinee, little reviewer experience. 

(10) "The pee^ review is basically sound. However, from time to 
time, problems arise because some reviewers' exhibit personal 
prejudices, particularly vis-a-vis competitors. At other times 
reviewers are inadequately informed or do not take the time to 
review the proposal adequately. This often results in ill- 
informed and hasty criticism and evaluation which has to be dealt 
with on an adversary basis. Other problems arise due to 
bureaucratic and political considerations within NSF. . . This is 
less frequent but when it happens it is difficult to deal with." 

— One-t/.me awardee with some reviewer experience. 



(11) "Peer review is necessary to maintain tne vitality of our 
national science effort. No other system seems to be obviously 
better or even as good. Problems with personal rivalries can be 
circumvented by a good program officer who knows the reviewers, 
and I have seen several examples . " 

— One-time awardee, considerable reviewer experience. 



(12) "I recognize that NSF cannot support all requests but it's 
frustrating when the reviews are mixed, and therefore the 
proposal's not funded. We don't all agree on what the important 
research is at the moment. Many of us like to think we are ahead 
and it hurts to be cut down by comments like 'not interesting' 
'not important'..." ' 

— Consistent declinee, moderate reviewer experience. 

(13) "I was quite surprised at the r ange of the reviews given my 
proposals. I don't know of any reasonable explanation for this 
lack of consistency, other than the fact that reviewers are human 
beings . " 

— Frequent declinee, no reviewer experience. 

(14) "I work long and hard to write a proposal. I do not 
appreciate a one paragraph review, with no technical reasons 
given to support the reviewer's unfavorable opinion... I feel if 
you are golrj to reject a proposal you should give good reasons, 
and it is the program director's responsibility to ensure these 
are given." 

— Consistent declinee, experienced reviewer. 
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(15) "When a proposal is rejected we get too little information 
on whether we should re-submit or not. Further^ we get only 
modest feedback on how a rejected proposal could profitably be 
modified. This leads to a continuous cycle of (often fruitless) 
resubmissions. Tell the PI if there is little hance of funding 
a similar proposal. It's better to be discouraged than to put a 
lot of effort in^ only to be rejected again". 

— Frequent declinee^ experienced reviewer. 



(16) "Program directors have an inordinate influence on the type 
of research funded. Influence is exercised through selection of 
reviewers and panelists and interpretation of reviews. As 
program directors come and go^ there are radical swings in what 
will and will not be funded." 

— Consistent awardee^ no reviewer experience. 



(17) "The peer review process has served our profession and our 

nation well Obviously^ it can be 'maneuvered' by program 

managers at NSF but I believe they do a good^ honest job and have 
made decisions which are correct and defensible." 

— Frequent awardee^ extensive reviewer experience. 



(18) "NSF gave up its autonomy and its ability to fund the 
cutting edge of research when it let reviewers' ratings count for 
mor^ than the program director's instinct and opinions. As a 
result, other funding agencies have, for years, openly said: 'If 
you have an original idea, bring it to us because NSF no longer 
funds new ideas ' — " 

— Frequent declinee, some reviewer experience. 
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(19) "Most program directors are quite good and are judicious in 
exercising authority to direct funding into certain areas. 
However, there have been certain situations where a program 
director was essentially lobbied by a colleague/ P.I. and funds 
redirected to that particular program. The assistant program 
directors have always , in my experience, been very professional 
in their conduct and have been most helpful." 

— Infrequent awardee, moderate reviewer experience. 

(20) "...given the interdisciplinary nature of my research, 
reviews of my proposals are often very narrowly conceived. 
Indeed, almost 40% of the reviews are critical of issues that are 
never raised or relevant to my proposals. I consider myself 
fortunate to have been treated fairly by the review panels. 
Nonetheless, I believe that the present system inhibits creative, 
multidisciplinary research." 

— Frequent awardee, extensive reviewer experience. 

?r 28 



(21) "Interdisciplinary programs are difficult to get funded 
because reviewers in each discipline criticize the other. 
Similarly, research proposals for work which does not fit into a 
particular program falls between the cracks and is difficult to 
get funded... Often the PI must make justifications in the 
proposals to make it appear that it fits. This weak link is 
frequently attacked by the reviewers..." 

— Frequent declinee, some reviewer experience. 



(22) "Several of my proposals were mis-directed (either by 
myself or the NSF routing procedure), m at least one case, a 
proposal that was finally funded by one program had been rejected"'' 
by another. This is despite the fact that the proposal was not 
changed and both NSF program panels had the same reviewers' 
comments to work from." 

— Frequent declinee, no reviewer experience. ' 



(23) "NSF has always been very open and helpful in guiding me to 
appropriate programs for my work. I think the 'interactiveness' 
of the programs helps a lot ".n establishing a feeling of 
confidence that the work will be dealt with fairly. My good 
fortune has been in contrast to many of my close colleagues 
(three in my department, who happen also to be women). I don't 
fully understand why this has happened." 

— Consistent awardee, little reviewer experience. 



(24) "As funding gets tighter, reviews get more critical and 
rivalry intensifies. This can easily go past the point where 
good science is singled it for funding, and can soon drive good 
people from the field. . This has happened in many fields, and is 
happening now in Britain. I see the seeds of a similar situation 
here ..." 

— Frequent awardee, moderate reviewer experience. 



(25) "Less than one proposal in ten is being funded by the 
program. . . The effect on established researchers is quite 
serious, but the impact on new young faculty is a disaster. . . If 
this situation does not improve soon, then we will lose an entire 
generation of the best minds and future teachers in our 
universities... In times such as the present, where the success 
rate on proposals becomes ridiculously low, the reviewers all 
become nervous and supercritical because they know that a 
positive review severely diminishes their own chances of funding. 
Thus peer review becomes cutthroat, much innovative work is not 
funded, and even very good people become frustrated..." 



Consistent awardee, extensive reviewer experience. 
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(26) "I submitted an earlier proposal and got good to excellent 
ratings. But the resubmitted proposal, based on reviewers* 
comments, fared much lower. Additionally, there was a clear 
implication that my association with a small, liberal arts 
college was a liability. . .It seems that I will never gain a grant 
from NSF because of my association with an * unknown', small 
institution. This is in spite of the fact that I publish 
regularly and am the editor of my professional journal." 

— Consistent declinee, little reviewer experience. 



(27) "The selection process at NSF is fair and efficient. 
However, I am very concerned about the limited support available 
for young investigators. I wish NSF would support new 
investigators to a larger extent, even though this means to take 
a great chance with respect to the success of the programs." 

— Frequent declinee, extensive reviewer experience. 



(28) "Greater emphasis should be placed on the track record, 
accomplishments, productivity of the principal investigator on 
previous grants. Too much time is spent in preparing proposals. 
This distracts form the performance of research. Older 
scientists should not be discriminated against if productivity is 
high." 

— Frequent awardee, experienced reviewer. 



(29) "The peer and panel review process is imperfect, but seems 
to be the best available. Narrowness and intolerance by the 
individuals involved is a problem. This is a fault in the way 
universities train students... Considerable effort is made to 
help beginning investigators, women, and minorities, faculty at 
undergraduate institutions, etc., and that is excellent. 
However, does it make sense to force out experienced 
investigators... some modest level of noncompetitive grants 
should be given, and all should pass through review at reasonable 
intervals. " 

— Frequent declinee, moderate reviewer experience. 
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(30) "Failure to gain significant amounts of grant support 
within the first few years of coming to my present position 
effectively ended all reasonable chance of ever gaining such 
support...! could not compete in productivity with those who had 
funds. Eventually, whatever reviewers might find meritorious in 
a proposal was countered by comments about low productivity. . .NSF 
should concentrate resources on making certain all young 
scientists in appropriate positions have an opportunity to 
initiate their studies. After a trial period continued support 
should be based on a reasonable level of productivity. Then we. 
could do away with most of this ridiculous paper shuffling, half- 
baked results rushed into proposals to please reviewers, etc." 

— Consistent declinee, no reviewer experience. 



(31) "NSF panels generally take chances on young researchers and' 
fund reliable superstars. The 75% of us inbetween these extremes 
are very likely to be dumped in mid-career by NSF." 

— Frequent declinee, moderate reviewer experience. 



(32) "After four years of NSF grants, being dropped had a 
devastating effect on my research. The preference toward juniors 
really depressed me... by penalizing quality researchers for 
falling from grace, the system discourages a lot of extremely 
radical ideas and directions." 

— Frequent declinee, no recent review experience. 



(33) "While exceedingly disappointed in the fate of my own 
submissions to NSF, I cannot imagine a better, more fair or more 
authoritative review process." 

— Consistent declinee, little reviewer experience. 



(34) "I can't imagine a process any better than peer review. It 
works in part because most reviewers take serious interest in 
their roles, in part because the people at NSF themselves are ' 
knowledgeable and of unquestioned integrity. It tends to break 
down when decisions are made on some basis outside of scientific 
merit (e.g., regional allocations of funds), are delayed or are 
not explained. " 

— Frequent declinee, moderate reviewer experience. 
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iX. Additional Topics 

A. Proportion of Time Devoted to Research 

Respondents were asked (Q. 33) to indicate roughly what 
proportion of their work time is devoted to various research, 
teaching and administrative activities. The applicant population 
as a whole divides its time as follows: 



Research, preparing proposals, related activities 41% 

Undergraduate instruction, preparation, advising 21% 

Graduate instruction, advising, consultation 18% 

Administrative duties to one's institution 11% 

Service to one's discipline (incl. as a reviewer) 6% 

Consulting or other outside employment 2% 

Other (unspecified) 2% 



(1) The greatest differences were between applicants from the two 
sets of Ph.D. -granting institutions and those from primarily 
undergraduate institutions, as illustrated by the following: 

(a) Time devoted to research and related activities: 

0-10% 11-20% 21-30% 31-40% 41% + 



"top 21" 4% 10% 22% 22% 42% 

"other Ph.D." 4% 12% 23% 20% 41% 

Prim. Undergrad. 22% 24% 23% 14% 17% 

(b) Time devoted to undergraduate instruction and related: 

0-10% 11-20% 21-30% 31-40% 41% + 

"top 21" 40% 29% 21% 6% 5% 

"other Ph.D." 34% 25% 24% 9% 9% 

Prim. Undergrad. 9% 8% 13% 14% 56% 



(2) By division: 50% or more of the applicants to five 
divisions (Cellular Biology, Molecular Biology, Atmospheric 
Sciences, Ocean Sciences and Physics) and 25% or fewer applicants 
to four other divisions (Computer Research and three engineering 
divisions) devote 41% or more of their time to research. 

(3) Little difference was reported among awardees and declinees 
and among persons with or without reviewer experience. Women 
reported devoting slightly more time to research and less to 
consulting and administrative work. 
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B. Factors Hindering Progress in Research 

In Q. 26, respondents were asked to indicate the degree to which 
each of six factors hindered progress in their own research. The 
factors and percent reporting them as a hindrance are: 

Shortages of capable graduate students 
Shortages of research personnel 
Inadequate equipment 
Inadequate facilities 
Unavailability of data 
Insufficient theoretical models 

(1) In each category, declinees were more likely to report such 
hindrances than awardees, and applicants from undergraduate 
institutions were more likely to report shortages in the first 
four categories than those from other classes of institutions. 

(2) Divisional percentages are fairly consistent with regard to 
shortages of graduate students, except that factor is cited more 
often by applicants to the Physics division and less often by 
applicants to Mathematical Sciences. 

(3) Over half the applicants to the Astronomical Sciences 
division and to the four divisions that handle biosciences 
proposals cite shortages of research personnel. This factor was 
cited by 25% or fewer of the applicants to the Divisions of Polar 
Programs and Mathematical Sciences. 

(4) Inadequate equipment is cited as a hindrance by more than 40% 
of the applicants to three engineering divisions, two geosciences 
divisions and three physical sciences divisions and by 15% or 
fewer applicants to the Mathematical Sciences and Social and 
Economic Sciences divisions. A similar pattern obtained for 
inadequacy of facilities. 

(5) Unavailability of data was cited more frequently by 
applicants to the Social and Economic Sciences, Information 
Sciences and Polar Programs Divisions and least cited by those to 
the physical sciences divisions. Insufficient theoretical models 
were cited more often than the divisional average by applicants 
to Social and Economic Sciences, Astronomical Sciences and two 
engineering divisions. 



47% 
46% 
36% 
29% 
17% 
14% 
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X. Limitations of the Data and the Analysis 

A. Nonrespondent bias: 

An issue that conunonly arises in surveys of this sort is whether 
nonrespohdents would have given the same replies as those who did 
respond. 88% of awardees responded and 52% of declinees (a much 
larger group )^ following one blind postcard reminder. We have 
chosen not to sample non-respondents^ for two reasons: 

(1) To encourage candid replies^ the survey forms were not coded 
or controlled to identify respondents. This renders controlled 
subsample selections impossible. 

(2) While it would be possible to resurvey all declinees (since 
we have records of who was declined in FY 1985)^ or a smaller 
group based on asking whether they had responded to the earlier 
survey^ the time and costs of doing so are substantial. 

B. Two-way vs. many-way analysis of variables: 

Most of the tables produced from the raw data are two-way tables^ 
with some exceptions (e.g.^ the categorization of respondents by 
experience class ^ which was constructed from several variables). 
Other sorts of analyses could be performed. The data is 
available for such purposes — see (1). 

C. State in which applicant's institution is located: 

We asked respondents to indicate the state in which their 
institution is located^ but decided not to undertake the complex 
regressions necessary to isolate answers by individual state or 
groupings; it is not clear that such analysis would be useful 
given the powerful relationship of attitudes to proposal/award 
experience^ research division, institution type and other 
factors. 



The survey was designed by Jim McCul lough. Bob Abel and Susan 
Queen of NSF's Program Evaluation Staff. The Center for Science 
and Technology Policy Studies of Abt Associates Inc. of 
(Cambridge, Mass.) undertook the mailings and compiled the data, 
under the direction of Stephen Fitzsimmons. Bob Abel developed 
computer programs and analyzed the data, with assistance from 
Bill Commins. Jim McCullough wrote the report, and Linda Rogers 
assisted with typing and preparation of appendices. The applicant 
classes are based on a suggestion by Carlos Kruytbosch of NSF's 
Science Resources Studies Division. 
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Notes : 



1. A set of detailed tables, and information about the 
availability of data, may be requested from: NSF Program 
Evaluation staff. Room 425, 1800 G St. N*W., Washington, D*C., 
20550 • 



2. Many proposals acted upon in FY 1985 were submitted before the 
start of the year; also, many persons submitting proposals during 
the year were not surveyed because the Foundation's decision 
about their proposal was made later. 

3. The complete list is shown in Q. 25 of Appendix G. 

'4. This included persons who applied through an independent 
research organization (e.g., SRI International, Carnegie 
Institution, Field Museum) or as individuals not associated with 
an institution. 

5. But placed in the current (FY 1987) Directorate matrix for 
clarity. Principal changes are the shift of the Divisions of 
Computer Research and Information Science and Technology to the 
since-established CISE Directorate, and the transfer of the 
Division of Astronomical Sciences to the MPS Directorate. 

6. Principally by adding "Materials Research" and "Information 
Science and Technology" as fields. 
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APPENDIX A: NSF PROPOSAL REVIEW AND DECISION MAKING PROCEDURES 



Most proposals received by the Foundation are unsolicited; the research 
ideas' and methods are those of the applicants. Proposals are assigned 
to a program officer who oversees external review, evaluates reviewers' 
comments, and makes a recommendation to award or decline it, taking 
into account other considerations — such as the relationship of the 
work to the field as a whole and to other pending proposals, and the 
program's purposes and budget. 

The program officer's recommendations are reviewed at one or more 
supervisory levels. In a few cases, they are further reviewed by the 
National Science Board. Specialists in the _ivision of Grants and 
Contracts then check on all the nonscientif ic aspects of the award and 
issue the formal notification to the applicant's institution. 

The specific review procedure depends on 'che tradition of the field: in 
the physical sciences, engineering, and some geosciences programs, 
proposals are usually mailed to reviewers who respond individually; in 
the biological and social sciences and some geosciences programs a 
panel reviews the proposal in addition to the mail revi3wers. For 
large projects and special competitions of various types reviews are 
often supplemented with site visits by teams of reviewers and staff. 

In the mail -out procedure, the proposal is sent to several people — 
sometimes as many as ten — identified by the program officer as 
knowledgeable on the topic. The reviewer receives standard 
instructions and forms and responds directly to the program officer. 
Where panels are used, they meet at some specified interval (generally 
three times a year in the larger programs) to weigh a group of 
proposals, taking into account the prior mail reviews. On average, a 
proposal has the benefit of 5.5 completed external reviews. 

The Foundation-wide criteria for proposal selection are published in 
NSF's "Grants for Scientific and Engineering Research and Education". 
They are meant to be applied to all proposals "in a balanced and 
judicious manner" : 

(1) Research performance competence; technical capability of 
investigators (s) and adequacy of institutional resources; 

(2) Intrinsic merit; extent to which the proposed work is 
expected to lead to new discoveries or fundamental advances in its 
field or across fields; 

(3) Utility or relevance; extent to which it could contribute to 
an extrinsic goal such as a new technology; and 

(4) Effects on the infrastructure of science and engineering: 
what the work will contribute to the Nation's research, education 
and human resource base. 
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The relative weights of criteria (2) and (3) depend on the character of 
the work. The fourth criterion allows the program to take into account 
such matters as the participation of women and minorities, 
institutional distribution, and the stimulation of important but 
underdeveloped research areas. In many cases programs also use special 
criteria related to their purpose as in, for example, programs for 
equipment grants or visiting professorships for women. 

Several safeguards are built in to ensure an open and fair system: 

(1) Conflicts of Interest; Reviewers and program officers are 
asked to identify any academic affiliations, personal 
relationships or financial interests that could constitute or be 
regarded as a conflict, and to withdraw from the decision process 
if warranted. 

(2) Verbatim Reviews; All reviews are routinely sent to the 
proposer, but without attribution to individual reviewers. Where 
panels are employed, summaries of their deliberations are also 
provided. 

(3) Fo rmal Reconsideration; At the applicant's request, 
reconsideration may be carried out by persons not involved in the 
earlier decision. This is not a "de novo" review of the 
scientific merits but an examination of how the case was handled 
by the program officer. An applicant not satisfied with the first 
reconsideration may have his or her institution request a further 
reconsideration by the Foundation's deputy director. 

(4) Sampling; The Director's Office of Audit and Oversight 
routinely samples actions for compliance with Foundation 
procedures and provides direct feedback to the program division. 

(5) External Peer Oversight; Each program is given a 
comprehensive review every three years by a small group of 
external peers familiar with the field. These "visiting 
committees" make formal reports to the NSF Dixictor. The 
reports, and the annual summary, are available to the public. 

(6) Analyses of Decision Patterns; The Director's Program 
Evaluation Staff runs statistical tests to see how the system has 
treated various classes of applicants. 



APPENDIX B; NSF RESEARCH SUPPORT ORGANIZATION 



NOTE: FY 1985 Divisions included in survey, placed in FY 1987 
Directorate Structure. 



Abbreviation Organization 

BBS Biological, Behavioral and Social Sciences Directorate 

BNS Behavioral and Neural Sciences 

BSR Biotic Systems and Resources 

DCB Cellular Biosciences 

DMB Molecular Biosciences 

SES Social and Economic Sciences 

CISE Computer and Information Science & Engineering 

Directorate 

DOR Computer Research 

1ST Information Science and Technology 

ENG Engineering Directorate 

CBT Chemical, Biochemical, and Thermal Engineering 

DMC Design, Manufacturing, and Computer- Integrated Eng. 

ECE Electrical, Communications, and Systems Engineering 

MSM Mechanics, Structures, and Materials Engineering 

ECES Emerging and Critical Engineering Systems 



Geosciences Directorate 

Atmospheric Sciences 
Polar Programs 
Earth Sciences 
Ocean Sciences 

Mathematical and Physical Science Directorate 

Astronomical Sciences 
Chemistry 

Materials Research 
Mathematical Sciences 
Physics 

Scientific, Technological and International Affairs 
Directorate 

Research Initiation and Improvement 



GEO 



ATM 
DPP 
EAR 
OCE 

MPS 

AST 
CHE 
DMR 
DMS 
PHY 

STIA 



RII 




APPENDIX C: 



Number of Proposals by Field of Research and by NSF Division 



Respondents* 


BBS 


BBS 


BBS 


BBS 


BBS 


CISE 


CISE 


ENG 


ENG 


ENG 


ENG 


ENG 


ENG 


GEO 


GEO 


GEO 


CEO 


HPS 


?1PS 


HPS 


NFS 


HPS 


STIA 


(2) 


Totals 


Field of Research 


BNS 


BSR 


DCB 


DNB 


SES 


OCR 


1ST 


CBTE 


DMCE 


ECES 


ECSE 


MSMP 


OTH 


ATM 


DPP 


EAR 


OCE 


AST 


CHEM 


DKR 


OHS 


PHY 


RII 


NONE 


by field 


Computer Science 








1 


2 


318 


38 




80 


2 


17 
















2 




11 


2 


12 


45 


531 


Chem . Eng . 






2 


6 




2 




304 




5 




9 


30 












11 


8 


1 




18 


29 


427 


Civil Eng. 


1 


1 




1 


5 






5 


12 


171 


2 


105 


24 






3 






2 


5 






3 


23 


364 


Elect. Eng. 


2 


1 




4 




10 




17 


44 


16 


268 


g 


27 


3 


2 








2 


10 


2 


2 


23 


34 


476 


Nech . Eng . 








2 








127 


31 


21 


9 


259 


19 


7 


4 


2 


3 


2 


2 


11 


1 




28 


35 


565 


Eng. Other (1) 


5 


2 


2 


4 


5 


9 


2 


56 


75 


108 


66 




33 


4 


2 




3 






18 


8 




11 


66 


592 


Atmospheric Sci. 




























145 


8 


3 


5 








2 






4 


170 


Earth Sci. 


8 


25 


1 


3 


11 










10 






2 


3 


57 


483 


39 












11 


39 


693 


Ocean Sci. 


4 


13 


1 


5 












2 




2 




10 


23 


10 


395 




3 


2 




2 


8 


22 


501 


Geo. Sci. Other (1) 


7 


11 


4 












-a 


14 






5 


17 


9 


62 


17 




2 










18 


177 


Information S&T 


1 








4 


3 


29 




























1 




8 




53 


Biochemistry 


31 


3 




339 


















o 






3 


3 




13 


2 


2 




13 


68 


608 


Cell/Molec. Bio. 


14 


18 


124 


119 






























4 








15 


20 


403 


?cology/Env. Bio. 


14 


224 


213 


4 


5 




















5 


4 


21 












10 


9 


302 


C4edical Sci. 


143 


6 


2 


22 


4 


2 












JL 


4 




5 




3 




3 


2 




2 


22 


50 


437 


Microbiology 


3 


22 


170 


66 








2 




5 








2 


4 




7 




2 








8 


12 


177 


Life Sci. Other (1) 


267 


482 


45 


229 


9 




7 


4 




11 


2 


2 






16 


7 


40 


2 


3 


1 


4 


3 


66 


131 


1678 


Psychology 


314 


2 


393 


1 


49 




11 






3 




2 






















16 


46 


449 


Zoology 


11 


51 


3 


3 


























3 












1 


14 


110 


Mathematics 


4 


1 


27 


1 


13 


16 


5 


2 


1 


2 


12 


7 




2 










3 


3 


966 


2 


23 


60 


1121 


Astroncmy 




























36 


3 






148 










1 


7 


197 


Chemistry 


30 


8 


2 


99 








34 




11 


2 


6 


12 


13 


5 


9 


4 




720 


115 


1 


8 


27 


153 


1268 


Materials Research 


6 




10 


2 








2 






3 


9 








1 






2 


143 




2 


8 


16 


194 


Physical Other (1) 


3 


2 




11 


2 


2 




4 




2 


3 




1 


28 


8 


96 


11 


14 


6 


15 


2 


7 




18 


242 


Physics 


5 




6 


10 


3 






17 




2 


30 


8 


7 


57 


16 


5 


5 


16 


22 


248 


8 


235 


8 


63 


770 


Anthropology 


266 


5 


5 


1 


3 
















2 






1 








1 






7 


9 


298 


Economics 


5 


3 


2 




287 




21 






















2 






2 






18 


338 


Soc. Sci. Other (1) 


99 


11 






242 




5 






5 






1 




2 


3 








2 


2 




46 


19 


437 


Political Science 


3 








132 










2 


























7 


12 


155 


Sociology 


3 


1 






124 




5 






3 


























24 


7 


168 


Misc. Other (1) 


38 


16 


5 


3 


18 




2 


2 






2 




1 


3 


2 


26 


5 


5 


1 


5 


1 




23 


14 


173 


Totals by Division 


1285 


907 


1015 


935 


929 


362 


128 


580 


248 


399 


418 


531 


170 


330 


172 


722 


564 


193 


799 


592 


1012 


263 


448 


1070 


14072 



Notes : 
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(1) Respondents did not identify specific field: also includes fields for which a small 
number of responses were given (e.g.. Aeronautical Engineering). 

(2) Program or division not reported or not identified. 
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Percent of Proposals that ^re Eventually Funded 

by Research Divisions appendix 
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^^^^^ BBS CISE ENG GEO MPS 
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Appendix E: Characteristics of ^plicants by Experience Class 





Sex 




Race 








M 


F 


W 


A 


M 


21 


One-Time Awardee: 


79% 


21% 


90% 


8% 


2% 


29% 


Frequent Awardee: 


89% 


11% 


88% 


10% 


2% 


28% 


consistent Awardee: 


94% 


7% 


92% 


7% 


2% 


37% 


One-Time Declinee: 


75% 


25% 


90% 


8% 


2% 


17% 


Frequent Dec! i nee: 


91% 


9% 


88% 


10% 


2% 


22% 


Consistent Declinee: 


87% 


13% 


88% 


10% 


2% 


14% 


Survey Universe: 


87% 


13% 


89% 


9% 


2% 


23% 
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Institution 




Professional Age 


Rev 


•r 


Ph.D. 


PUI 


Other 


>59 


60s 


70s 


80s 


Y 


N 


52% 


12% 


7% 


12% 


22% 


37% 


29% 


80% 


20% 


58% 


8% 


6% 


11% 


28% 


43% 


16% 


91% 


9% 


54% 


4% 


5% 


19% 


35% 


37% 


8% 


97% 


3% 


61% 


18% 


5% 


12% 


23% 


37% 


29% 


44% 


56% 


65% 


9% 


4% 


9% 


26% 


45% 


20% 


83% 


17% 


68% 


14% 


4% 


8% 


25% 


43% 


24% 


62% 


38% 


62% 


11% 


4% 


11% 


26% 


41% 


21% 


75% 


25% 
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APPENDIX F: extent OF PRE-PROPOSAE. CONTACT 



Overall 



Award 
Decline 

Experience with NSF 
ITime Awardee 
Freq. Awardee 
Cons. Awardee 
ITime Declinee 
Frequent Declinee 35 
Cons. Declinee 



Division 
BBS-BNS 
BBS-BSR 
BBS-DCB 
BBS-DMB 
BBS-SES 
CISE-DCR 
CISE-IST 
ENG-CBTE 
ENG-DMCE 
ENG-ECKS 
ENG-ECSE 
ENG-MSME 
ENG-OTHER 
GEO-ATM 
GEO-DPP 
GEO-EAR 
GEO-OCE 
MPS-AST 
MPS-CHEM 
MPS-DMR 
MPS-DMS 
MPS- PHY 
STIA-RII 
NONE-OTHER 



(1) 


(2) 


(3) 


(4) 


(5) 


(6) 


(7) 


(8) 


33% 


21% 


52% 


13% 


5% 


26% 


5% 


25% 


32 


23 


58 


16 


6 


37 


4 


20 


34 


20 


49 


11 


5 


21 


5 


27 


32 


21 


52 


13 


5 


25 


5 , 


24 


71 

.3 ± 




o 1 


19 


5 


42 


5 


18 


29 


23 


59 


18 


6 


48 


4 


19 


37 


22 


46 


5 


4 


10 


5 


29 


35 


19 


54 


14 


6 


28 


3 


24 


35 


18 


46 


10 


6 


15 


4 


30 



^8 


26 


57 


6 


3 


26 


4 


25 




22 


58 


14 


5 


21 


3 


23 


a{ 


lo 


53 


7 


3 


11 


1 


30 


32 


17 


47 


8 


2 


14 


2 


32 


35 


31 


56 


12 


4 


28 


8 


21 


32 


18 


38 


13 


3 


28 


2 


37 


50 


31 


64 


24 


4 


20 


17 


10 


26 


16 


47 


15 


7 


30 


4 


22 


37 


16 


46 


26 


7 


24 


5 


24 


32 


21 


53 


27 


9 


39 


8 


17 


27 


18 


42 


20 


7 


22 


5 


31 


34 


13 


46 


22 


3 


22 


4 


28 


46 


13 


35 


18 


5 


20 


5 


22 


17 


28 


73 


25 


3 


55 


9 


10 


34 


28 


70 


21 


5 


44 


5 


9 


30 


16 


56 


8 


5 


33 


2 


27 


26 


20 


64 


23 


9 


55 


6 


16 


20 


19 


64 


17 


4 


47 


4 


10 


40 


23 


53 


13 


7 


36 


3 


20 


33 


23 


68 


20 


4 


33 


4 


10 


28 


13 


31 


4 


8 


21 


1 


51 


21 


26 


68 


21 


3 


41 


6 


13 


66 


25 


49 


7 


20 


12 


20 


20 


47 


22 


49 


10 


4 


13 


4 


24 



ERIC 



Year of Highest 


Degree 
















1959/before 


30 


24 


49 


17 


3 


36 


6 


21 


1960-69 


32 


21 


53 


17 


5 


33 


5 


22 


1970-79 


35 


21 


55 


12 


6 


25 


5 


24 


1980-87 


34 


17 


48 


9 


6 


15 


3 


32 


Male 


32 


21 


52 


14 


5 


27 


5 


25 


Female 


40 


22 


56 


9 


6 


21 


4 


23 


Resubmission 


33 


24 


59 


16 


6 


27 


5 


21 


1st Submission 


34 


20 


50 


12 


5 


26 


4 


26 


Institution 


















"Top 21" 


27 


19 


50 


13 


4 


35 


5 


28 


"Phd-Other" 


32 


20 


52 


13 


5 


25 


4 


26 


Undgrad 


52 


26 


56 


7 


8 


14 


5 


16 


Other 


38 


25 


57 


22 


5 


28 


9 


15 


Race/Ethnicity 


















Asian 


28 


17 


40 


12 


5 


16 


3 


35 


Minority 


34 


20 


49 


15 


14 


20 


3 


20 


White 


34 


21 


54 


13 


5 


28 


5 


24 


Reviewer 


32 


21 


55 


15 


5 


33 


5 


23 


Not Reviewer 


39 


20 


46 


7 


5 


7 


4 


30 



KEYS 
(1) 
(2) 
(3) 
(4) 



Received NSF mailing (5) 

Wrote NSF (6) 

Telephoned NSF (7) 

Visited NSF (8) 



NSF Visited Institution 
Knew Program officer 
Submitted Preliminary Proposal 
Institution Handled all Contacts 
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NATIONAL SCIENCE FOUNDATION 



WASHINGTON, D.C. 



20550 




APPENDIX G: SURVEY FORM 



OFFICE OF THE 
DIRECTOR 



October 31, 1986 



Dear Colleague: 

We .at the Foundation value the views of the communities we serve 
and make every effort to take them into account in designing and 
operating our programs. 

We are now surveying investigators whose proposals for research 
support were awarded or declined last year, to see how well our 
proposal review system is working and whether information about 
partfc^lar programs is reaching potential applicants. 

Please take twenty or thirty minutes to complete the enclosed 
survey and return it in the postpaid envelope. A few of the 
questions solicit your written comments, and I encourage you to 
provide them. Survey results will be presented to me, to the 
National Science Board, and to Foundation managers and staff as 
we consider ways to improve our programs and management systems. 

I hope you will not hesitate to answer frankly and thoughtfully. 
The survey is. designed so that individual responses cannot be 
associated with any person, institution or proposal. Replies 
will be processed by an organization outside NSF reporting 
directly to the Program Evaluation Staff , a part of my office. 
NSF program directors and other audiences will receive general, 
summarized information but will, not see your reply or those of 
any other respondent. 

Thank you for taking part in this effort. I am certain that the 
information from this survey will be useful to the Foundation, 
and I look forward to hearing from you. 



Erich Bli 
Director 
















0MB No. 3145-0098 
Expiration Date: 1/31/87 








SURVEY OF NSF RESEARCH APPLICANTS 










You have been selected for this survey because you subDltted a 
research proposal to NSF that was awarded or declined between 
October 1, 1984 and September 30, 1985. We would like to know your 
opinions about the procedures used to review, that proposal. If you 
had more than one proposal awarded or declined during that time, 
please answer the following questions with regard to the last of 
those proposals about which a decision was made. 


• 






1. Was the proposal that you submitted: CIRCLE ONE 










A new proposal for work not currently supported 




80% 






A proposal for further funding of ongoing NSF- 


>,2 


20% 






2. Was the proposal a modification of a proposal previously 
declined by NSF? 














29% 










71% 






3. Was this proposril principally for suooort ofj CTRni.P. hnp. 














92% 










5% 






Indus try- University Cfioperative Research 


..3 


1% 






Otner, PLEASE SPECIFY 


4 


1% 


; 




4. Was this proposal for support of multidisciplinary research? 














29% 




j 






71% 




f 
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5. Was this the first NSF proposal for which you served as the 
i-rincipal investigator? CIRCLE ONE 

1 29% 

NO. 2 71% 

6. Which NSF program handled your pjcposal? 

(If program unknown, speciTy NS,' division or directorate) 

Progra m 

7. How were you in contact with NSF prior to submitting your 
proposal? PLEASE CIRCLE ALL THAT APPLY. 

Received an NSF mailing 1 33% 

Wrote to NSF 2 21% 

Phoned NSF 3 52% 

Visited NSF 4 12% 

NSF came to my institution with a 
presentation 5 5% 

Knew a program officer 6 26% 

Sent in a preliminary informal proposal 7 4% 

Other, PLEASE SPECIFY^ ^8 2% 

My institution's research support office 
handled all contacts 9 25% 

8« What was the amount of money requested in your proposal 
on a yearly basis , including indirect costs? CIRCLE ONE 

Less than $30)000 « 1 13% 

$30,000 - $49i999 2 ^1% 

$50,000 - $69,999 3 22% 

$70,000 - $99,999 4 23% 

$100,000 - $149,999 5 

Over $150,000 6 7% 
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9. Which of the following review procedures were used for that 
proposal? PLEASE CIRCLE ALL THAT APPLY* 

Mail Review 1 

panel Review 2 

Pre-Award Site Visit 3 

Don' t Know 4 



7655 
42% 

1% 
16% 



10. Approximately how many months elapsed from when you or your 
institution sent the proposal to NSF until you received 
the Foundation's official letter conveying its decision? 
ENTER NUMBER OF MONTHS. 

Months Average 
25th p ercent i le 
75th percentile 

!!• Was processing of your proposal delayed because it lacked 
information required by NSF such as signatures or the 
statement about current and pending support? 



Yes, 



7.4 months 
6 months 
9 months 



1% 



Don' t know , 3 



12. 



Did you receive informal notification of the decision on 
your proposal before you received tho official notification 
letter? 



95% 
4% 



Yes, 
NO. . 



. 1 

.2 



56% 
44% 
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13« How ouch time elapsed from the time you learned of the 

decision on your proposal u'atll you received the reviewers' 
coeaents? CIRCLE ONE 

Received reviewers' comments at same time 



I learned of the decision 1 

One month or less < 2 

More than one Month (ENTER THE 
NUMBER OF MONTHS). Months 3 

Did not receive reviewers' comments ••••#••••4 



14* If you summed up the rating of your proposal by the 
reviewers^ which of the following categories would 
come closest to describing their overall rating? 



CIRCLE ONE 

Excellent • • • ^ 1 

Very Good to Excellent 2 

Very Good 3 

Good to Very Goo4 ^ 

Good •••••••5 

Fair to Good •••••6 

Fair or Lower ••••7 



4S 



15« Was the proposal you submitted to NSF awarded or was 
it declined? CIRCLE ONE 



Awarded (SKIP TO Q* 19) 1 

Declined • • • • • 2 

I 
I 



I 



I15A. What did you subsequently do with your 
proposal? CHECK ALL THAT APPLY 

Resubmitted to NSF with substantial revision 1 

Resubmitted to NSF with little or no 
revision 2 

Submitted it to another f unding source with 
substantial revision •••••3 

Submitted i^ to another funding source with 
little or no revision*..*. « 4 

Did not take further action on the proposals 3 

Other, PLEASE SPECIFY 6 



I 



34% 
66% 



To what degree did the reviewers' comments 
iul'luunce your decision regarding what ycu 
did with your proposal? CIRCLE ONE. 

Substantially e... 1 

Slightiy o 2 

Not at all...., ^ ..3 

Did not rec iive comments 4 
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16* If your proposal was declined, do you think the decision 
was made fairly? 



Yes->Hade Fairly I 40% 

No-Not Made Fairly 2 

I 

I 

I I6A« Why do you think that the decision was not | 
I made fairly? | 



( 96% of those who indicated "not fairly" 



stated their reasons ) | 

I 

I 

I 

I 



I7« If your proposal was declined, did you contact the program 



officer to discuss the reasons? 

Yes 1 47% 

NO 2 53% 

18. Were you aware that NSF has a formal reconsideration process? 

Was aware (SKIP TO Q. 21} I 18% 

Was not aware (SKIP TO Q. 2I).... 2 82% 

I9« If your proposal was awarded, to what extent did the 
reviewers' comments influence the research that was 
subsequently conducted? CIRCLE ONE 

Substantial influence I 5% 

Some influence * 2 38% 

Little influence 3 34% 

No influence •A £3% 

Did not receive comments 5 5% 



ERJC 
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20* Would you say that all or much of the research carried 
out under the NSF award has **applied", "practical** or 
"policy** implications beyond the advancement of 
knowledge itself? 

Yes (ANSWER Q. 20A) 1 

NO (SKIP TO Q. 21) 2 



I 

20A. If yes, are these implications: | 

I 

Clear and immediate o 1 I 

Fairly clear, long range 2 I 

Potential but not clear • 3 I 

, I 



21. Whether your proposal was awarded or declined, how much 
did the reviewers' comments help you to understand the 
Foundation's decision? CIRCLE ONE 

A great deal » 1 

Somewhat ••2 

Slightly 3 

Not at all A 

22* Whether your proposal was awarded o"* declined, how satisfied 
or dissatisfied are you with the review process overall as it 
operates at NSF? CIRCLE ONE 

Very satisfied 1 

Moderately satisfied 2 

Neither satisfied nor dissatisfied....* 3 

Moderately dissatisfied (ANSWER Q. 22A) 4 

Dissatisfied (ANSWER Q. 22A) 5 



I 

22A. Why are you dissatisfied? I 

(93% Stated reasons) I 

I 

I 

\ 



23. Are yoi: 



Male (SKIP TO Q* 24) i 87% 

Female 2 13% 

I 

I 



2 3 A* If eligible*, did you submit your proposal to the f 

NSF Research Opportunities for Women activity? | 

Yes (SKIP TO Q. 24) 1 t ' 31% 

NO 2 I 69% 

I I 

I I 

I I 

23B. Were you aware of the NSF Research Opportunities | 

for Women activity that is designed to promote | 

opportunities for women scientists and engineers to | 

conduct independent research? I 

Yes ( *;NSW£R Q. 23C) 1 | 45% 

NO (SKIP TO Q. 24) 2 ! 55% 



23C. If yes, please tell us why you did 'not submit 
your proposal through this program. 
(M% nf thn<;p whn wprp awarp of thp program hnf. 

did not submit their proposal to the program 

stated reasons) 



*Women eligible to submit proposals are: 

Those who have received their doctorates at least 3 years 
prior to submission of the proposal to NSF and have not 
previously served as a principal investigator on an 
individual Federal award for scientific or engineering 
research; or those with doctorates whose subsequent research 
careers have been interrupted for at least 2 or the past 5 
years and who have not served as a principal investigator on 
a Federal award for scientific or engineering research since 
reentering their careers. 



03 



24, Is your race/ethnicity; 

White (SK-I-P TO Q. 25) 1 

Asian (SKIP TO Q. 25) 2 

Black...: 3 

Hispanic 4 

Native American «5 



88% 
9% 
.8% 
1.1% 
.2% 



T 



2UA. 



If eligible*, did you submit this proposal to 
the Minority Research Initiation program at NSF? 

Yes (SKIP TO Q. 25) 1 



No. 



24B. 



Were you aware of the Minority Research 
Initiation program at NSF to increase the 
participation of minority researchers? 



Yes 

No (SKIP TO Q. 25). 



,1 
.2 



24c. 



If yes, please tell us why you did not submit 
your proposal through this program. 

(77% stated reasons) 



'Persons eligible to submit proposals are: 

Minority scientists and engineers who hold full-time faculty or 
research-related positions at colleges or universities in the 
United States, its possessions and territories, and have not 
previously received Federal research support as faculty members. 

The term "minority" refers to those ethnic minority groups that 
are significantly underreprcrented in advanced levels of science 
and engineering, i.e., Blacks, Native Americans, Mexican 
Americans, Puerto Ricans, Alaskan Natives (Eskimo or Aleut), and 
Native Pacific Islanders (Polynesian or Microne;.ian) . 
Investigators must be nationals of the United States. 



24% 
76% 



41% 

59% 



n 



J' 



> 
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25« Was your institution: 

One of the following institutions that received sixty or more 
NSF research project grants in 1985 (SKIP TO Q. 26) I 



Columbia Univ* 


U. 


of 


Califoruia - Berkeley 


Cornell Univ. 


U. 


of 


Cnlifornia - Los Angeles 


Harvard Univ • 


U. 


of 


California - San Diego 


Massachusetts Inst • of 


U. 


of 


Colorado - Boulder 


Technology 


U. 


of 


Illinois - Urbana 


Northwestern Univ • 


U. 


of 


Maryland - College Park 


Ohio State Univ. 


U. 


of 


Michigan - Ann Arbor 


Princeton Univ. 


U. 


of 


Pennsylvania 


Purdue Univ. 


U. 


of 


Texas Austin 


Stanford Univ. 


U. 


of 


Washington 




U. 


of 


Wisconsin - Madison 




Woods 


Hole Oceanographic inst . 



Other University that offers doctoral degrees (SKIP TO Q. 26). ...2 

Undergraduate or Primarily Undergraduate 
Institution* (ANSW!2R Q. 25A) 3 

Other (SKIP TO Q. 26) 4 

I 25A. Did you submit your proposal for review through the 
I Progr-'tii for Primarily Undergraduate Institutions? 

I Yes (SKIP TO Q. 26) I 

I No • 2 

I I 

I 25B. Were you aware that NSF has a program designed for pro- 

I posals submitted from primarily undergraduate institutions? 

I Yes I 

I No (SKIP TO Q. 26) 2 



25C. If yes, please tell us why you did not submit your 
proposal through this program. 

(82% Stated reasons) 



*NSF defines primarily undergraduate institutions as those 
institutions that did no*, award more than 20 doctorates in 
fields of science/engineering supported by NSF in the 2 calendar 
years preceding the proposal submission date. 
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To what extent is progress in your own research work hindered 
because of: 

CIRCLE ONE NUMBER ON EACH LINE 



(N/A » Not Applicable) 



< — Not at All 



A* Shortages of research personnel 

Shortages of capable graduate students* • 

C. Unavailability of data 

D. Insufficient theoretical models 

S* Inadequate equipment • 

F. Inadequate facilities 

G. Other, PLEASE SPECIFY^ 



To a Great — > 
Extent 



2 


3 


4 


5 


N/A 


2 




4 


5 


N/A 


2 


3 


4 


5 


N/A 


2 


3 


4 


5 


N/A 


2 


3 


4 


5 


N/A 


2 


3 


4 


5 


N/A 



AVERAGE 

3ol 
3.2 
2.1 
2.0 
2.8 
2.6 



11. To what extent do you agree or disagree with the following 
statements . 

CIRCLE ONE NUMBER ON EACH LINE 



< — STRONGLY 
DISAGREE 



STRONGLY~> 
AGREE 



A. The present peer review process at NSF 
makes it difficult for researchers to 
submit 'mult idisciplinary proposals 



.1 2 



AVERAG E 




B. 



C. 



D. 



NSF should make grer.ter use of grants 
that permit groups of researchers to 
band together in pursuing common 

research problems 4 1 2 

NSF is not likely to fund high-risk 

exploratory research because the 

likelihood of obtaining favorable 

reviews is slim I 2 

In making awards NS^ should place 

substantially more emphasis on the 

research history of the investigator 

except for young researchers 1 2 



3.1 



4,0 



3.4 



E* Proven researchers experience 
difficulty obtaining NSF awards 
when they apply for grants outside 
their disciplines , 



1 2 



4.0 



28* Referring to the institutional categories listed in question 
25, from what type of institution did you receive your 
highest degree? CIRCLE ONE 

One of the institutions listed in 
Question 25 1 45% 



Other University that offers doctoral 
degrees , 2 

Undergraduate or primarily Undergraduate 
Institution 3 



54% 
4% 



29. From what type of institution did you receive your B.S. or 
B.A. degree? CIRCLE ONE 

One of the institutions listed in 

Question 25 1 22% 

Other University that offers doctoral 
degrees 2 49% 

Undergraduate or primarily Undergraduate 
Institution 3 £9% 

30. What is your principal field of research? (e.g. Chemistry, 
Physics, Oceanography, Mechanical Engineering, etc.) ENTER FIELD 

Field Over 97% reported their field. 

31. In wh;ic state is your present institution located? ENTER STATE 

stateCA 11%, NY 10%, MA 6%, TX 5%, OH 4%, MI 3.5%, FL 3% 



32. What year did you receive your highest degree? ENTER 'EAR 

Year 

before 1960 11% 

1960-1969 25% 

1970-1979 41% 

1980-1987 21% 




33. In a normal week, roughly what proportion of your actual 
work time is devoted to these activities (to nearest 5%^: 



Undergraduate instruction, preparation, advising 21 



Graduate teaching, advising, consultation. 


etc. 


1ft 


% 


Research and related activities, preparing 


proposals 


41 


7. 


Service to your discipline: e.g. reviewing 
proposals 




fi 


X 


Consulting or other outside employment 




2 


% 


Administrative duties for your institution 




11 


% 


Other 




2 


% 



Total - 100%* 



34. Approximately how many articles have you had published in lesS than 2 8% 

referred journals over the last five years? 3«5 ]^g«^ 

Number of Articles j^g ^^cg 

35. Approximately how many graduate students (M.S. or Ph.D.) have 

you supervised over the last five years? 0 15% 

1-2 19% 

Number of Graduate Students 3-5 29% 

6-9 17% 

j^Q^ ]^g<^ 

36. Approximately how many post-doctoral researchers have you 

trained over the past five years? q g^^^ 

1 1 wi% 

Number of Pos t-doctorals I zZ 

' 2 9% 

3-4 8% 

37. During the past five years, have you served as a mail reviewer 5.....»., 7% 
or panelist for NSF? 

Yes (Answer Question 37A) 1 

No (Skip to Question 38) 2 25% 



37A. During the past five years, for how many proposals 
have you completed written reviews? 



Number of Proposals 



1... 




2... 


9% 


3... 


10% 


4-6... 


24% 


7-10.. 


19% 


11-19., 


13% 


20+.. 


......17% 



^8 



38. Approximately how many proposals have you submitted to NSF during the 1-5 

past five years? ENTER NUMBERS OF PROPOSALS ; TnJ 

6-10#.. 10% 

39. Approximately how many of the above proposals were funded bv NSF? 
ENTER NUMBER OF NSF AWARDS ^ 



(See p. 9 of report text) 

40. Of the number submitted to but not funded by NSFr approximately 

how many were funded by other sources? ENTER NUMBER FUNDED O,,,, 64% 

Number Funded by Other Sources 

2« • • • 8% 

41. What is the current level of funding for your research whether as PI or 3+,.. 4% 
researcher from all extramural sources on a yearly basis ? (ROUND TO 

NEAREST $lrOOO. FOR EXAMPLE: ENTER 50 FOR $50^000) 

less than lOK 24% 
Level of Funding 10K-39K 20% 

42. Which Federal Agency (including NSF) or other funding source is you^7nD""i^Qi/* * 
first choice for support of your research? ENTER FUNDING SOURCE /0K-149K. . ^u>, 

150K+. 19% 

Agency or Funding Source. > NSF, . .66%. NIH..>20%, O ther 19% 
42A. Why? (84% Stated reasons) 



43. We would welcome any comments that you might have regarding 
the selection process of proposals as it operates at NSF. 
Please attach additional sheets of paper if necessary. 



(50% provided comments) 
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